Skip to content

Add option to handle watchdog alerts#124

Open
martialblog wants to merge 2 commits intomainfrom
watchdogs
Open

Add option to handle watchdog alerts#124
martialblog wants to merge 2 commits intomainfrom
watchdogs

Conversation

@martialblog
Copy link
Member

@martialblog martialblog commented Mar 12, 2026

In Prometheus a "watchdog" or "dead man's switch" is an alert that is always firing to ensure alerting pipeline is working. The -W, --watchdog flag can be used to flip/negate the exit state of the plugin for these kind of alerts:

$ check_prometheus alert --name Watchdog -W --no-alerts-state 2
[OK] - 1 Alerts: 1 Firing - 0 Pending - 0 Inactive
\_ [OK] [Watchdog] is firing - value: 1.00 - {"alertname":"Watchdog","severity":"none"}
|total=1 firing=1 pending=0 inactive=0
$ check_prometheus alert --name Watchdog -W --no-alerts-state 2
[CRITICAL] - 0 Alerts: 0 Firing - 0 Pending - 0 Inactive
\_ [CRITICAL] No alerts retrieved
|total=0 firing=0 pending=0 inactive=0

Fixes #123

See also #61 (comment)

@martialblog martialblog added this to the v0.7.0 milestone Mar 12, 2026
@martialblog martialblog self-assigned this Mar 12, 2026
@martialblog
Copy link
Member Author

@wattebausch @gianmarco-mameli

I'm planing to add this feature in the next release. Any feedback and testing is welcome.

Regards
Markus

@gianmarco-mameli
Copy link
Contributor

Hi @martialblog, I don't have this kind of alerts but I try to find some time to manage a test

@wattebausch
Copy link

@martialblog i re-enabled the watchdog on my kube-prometheus-stack and it works like a charm

$ ./check_prometheus -H test.example.org -p 443 -s alert -P -W
[OK] - 1 Alerts: 1 Firing - 0 Pending - 0 Inactive
\_ [OK] [Watchdog] is firing - value: 1.00 - {"alertname":"Watchdog","severity":"none"}
|total=1 firing=1 pending=0 inactive=0

$ ./check_prometheus -H test.example.org -p 443 -s alert -P 
[CRITICAL] - 1 Alerts: 1 Firing - 0 Pending - 0 Inactive
\_ [CRITICAL] [Watchdog] is firing - value: 1.00 - {"alertname":"Watchdog","severity":"none"}
|total=1 firing=1 pending=0 inactive=0

many thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Add option to change or flip the exit codes for alert subcommand

3 participants